Python Machine Learning 2nd Edition by Sebastian Raschka and Vahid Mirjalili, Packt Publishing Ltd. 2017

Code Repository: https://github.com/rasbt/python-machine-learning-book-2nd-edition

Code License: MIT License

Python Machine Learning - Code Examples

Chapter 16 - Modeling Sequential Data Using Recurrent Neural Networks

Note that the optional watermark extension is a small IPython notebook plugin that is being used to make the code reproducible. You can just skip the following line(s).

In [1]:
%load_ext watermark
%watermark -a "Sebastian Raschka & Vahid Mirjalili" -u -d -v -p numpy,scipy,pyprind,tensorflow
Sebastian Raschka & Vahid Mirjalili 
last updated: 2017-09-11 

CPython 3.6.1
IPython 6.1.0

numpy 1.12.1
scipy 0.19.1
pyprind 2.11.1
tensorflow 1.3.0
In [1]:
%load_ext watermark
%watermark -a "Sebastian Raschka & Vahid Mirjalili" -u -d -v -p numpy,scipy,pyprind,tensorflow
Sebastian Raschka & Vahid Mirjalili 
last updated: 2019-11-13 

CPython 3.6.9
IPython 7.8.0

numpy 1.16.4
scipy 1.3.1
pyprind not installed
tensorflow 1.14.0

The use of watermark is optional. You can install this IPython extension via "pip install watermark". For more information, please see: https://github.com/rasbt/watermark.

In [1]:
from IPython.display import Image
In [2]:
import gzip


with gzip.open('movie_data.csv.gz') as f_in, open('movie_data.csv', 'wb') as f_out:
    f_out.writelines(f_in)

Introducing sequential data

Modeling sequential data: Order matters

Representing sequences

In [3]:
Image(filename='images/16_01.png', width=600) 
Out[3]:

Understanding the different categories of sequence modeling

In [4]:
Image(filename='images/16_02.png', width=600) 
Out[4]:

Recurrent neural networks for modeling sequences

Understanding the structure and flow of a recurrent neural network

In [5]:
Image(filename='images/16_03.png', width=600) 
Out[5]:
In [6]:
Image(filename='images/16_04.png', width=600) 
Out[6]:

Computing activations in an RNN

In [7]:
Image(filename='images/16_05.png', width=600) 
Out[7]:
In [8]:
Image(filename='images/16_06.png', width=600) 
Out[8]:

The challenges of learning long-range interactions

In [9]:
Image(filename='images/16_07.png', width=600) 
Out[9]:

Long short-term memory units

In [10]:
Image(filename='images/16_08.png', width=600) 
Out[10]:

Implementing a multilayer RNN for sequence modeling in TensorFlow

Performing sentiment analysis of IMDb movie reviews using multilayer RNNs

Preparing the data

In [11]:
Image(filename='images/16_09.png', width=600) 
Out[11]:
In [1]:
import pyprind
import pandas as pd
from string import punctuation
import re
import numpy as np


df = pd.read_csv('2columnsSpencer.csv', encoding='utf-8')
print(df.head(3))
                         review  sentiment
0                  missed L jab          0
1  missed R cross, missed L jab          0
2                  missed L jab          0
In [2]:
df.shape
Out[2]:
(497, 2)
In [3]:
type(df['review'])
Out[3]:
pandas.core.series.Series
In [3]:
## @Readers: PLEASE IGNORE THIS CELL
##
## This cell is meant to shrink the
## dataset when this notebook is run 
## on the Travis Continuous Integration
## platform to test the code as well as
## speeding up the run using a smaller
## dataset for debugging

import os


if 'TRAVIS' in os.environ:
    df = pd.read_csv('movie_data.csv', encoding='utf-8', nrows=500)
In [4]:
## Preprocessing the data:
## Separate words and 
## count each word's occurrence


from collections import Counter


counts = Counter()
pbar = pyprind.ProgBar(len(df['review']),
                       title='Counting words occurences')
for i,review in enumerate(df['review']):
    text = ''.join([c if c not in punctuation else ' '+c+' ' \
                    for c in review]).lower()
    df.loc[i,'review'] = text
    pbar.update()
    counts.update(text.split())
Counting words occurences
0% [##############################] 100% | ETA: 00:00:00
Total time elapsed: 00:00:01
In [5]:
## Create a mapping:
## Map each unique word to an integer

word_counts = sorted(counts, key=counts.get, reverse=True)
print(word_counts[:5])
word_to_int = {word: ii for ii, word in enumerate(word_counts, 1)}


mapped_reviews = []
pbar = pyprind.ProgBar(len(df['review']),
                       title='Map reviews to ints')
for review in df['review']:
    mapped_reviews.append([word_to_int[word] for word in review.split()])
    pbar.update()
Map reviews to ints
['hold', 'holding', 'continues', 'r', 'and']
0% [##############################] 100% | ETA: 00:00:00
Total time elapsed: 00:00:00
In [6]:
len(mapped_reviews)
Out[6]:
497
In [7]:
sequence_length = 200  ## sequence length (or T in our formulas)
sequences = np.zeros((len(mapped_reviews), sequence_length), dtype=int)
for i, row in enumerate(mapped_reviews):
    review_arr = np.array(row)
    sequences[i, -len(row):] = review_arr[-sequence_length:]
In [8]:
sequences.shape
Out[8]:
(497, 200)
In [24]:
## Define fixed-length sequences:
## Use the last 200 elements of each sequence
## if sequence length < 200: left-pad with zeros

sequence_length = 200  ## sequence length (or T in our formulas)
sequences = np.zeros((len(mapped_reviews), sequence_length), dtype=int)
for i, row in enumerate(mapped_reviews):
    review_arr = np.array(row)
    sequences[i, -len(row):] = review_arr[-sequence_length:]

X_train = sequences[:347, :] # must make testing set divide evenly by the batch_size from train() built later
y_train = df.loc[:347, 'sentiment'].values
X_test = sequences[347:, :]
y_test = df.loc[347:, 'sentiment'].values


np.random.seed(123) # for reproducibility

## Function to generate minibatches:
def create_batch_generator(x, y=None, batch_size=64):
    n_batches = len(x)//batch_size
    x= x[:n_batches*batch_size]
    if y is not None:
        y = y[:n_batches*batch_size]
    for ii in range(0, len(x), batch_size):
        if y is not None:
            yield x[ii:ii+batch_size], y[ii:ii+batch_size]
        else:
            yield x[ii:ii+batch_size]
In [25]:
## @Readers: PLEASE IGNORE THIS CELL
##
## This cell is meant to shrink the
## dataset when this notebook is run 
## on the Travis Continuous Integration
## platform to test the code as well as
## speeding up the run using a smaller
## dataset for debugging

#if 'TRAVIS' in os.environ:
  #  X_train = sequences[:250, :]
   # y_train = df.loc[:250, 'sentiment'].values
    #X_test = sequences[250:500, :]
    #y_test = df.loc[250:500, 'sentiment'].values

Embedding

In [17]:
Image(filename='images/16_10.png', width=600) 
Out[17]:

Building the RNN model

In [26]:
import tensorflow as tf


class SentimentRNN(object):
    def __init__(self, n_words, seq_len=200,
                 lstm_size=256, num_layers=1, batch_size=64,
                 learning_rate=0.0001, embed_size=200):
        self.n_words = n_words
        self.seq_len = seq_len
        self.lstm_size = lstm_size   ## number of hidden units
        self.num_layers = num_layers
        self.batch_size = batch_size
        self.learning_rate = learning_rate
        self.embed_size = embed_size

        self.g = tf.Graph()
        with self.g.as_default():
            tf.set_random_seed(123)
            self.build()
            self.saver = tf.train.Saver()
            self.init_op = tf.global_variables_initializer()

    def build(self):
        ## Define the placeholders
        tf_x = tf.placeholder(tf.int32,
                    shape=(self.batch_size, self.seq_len),
                    name='tf_x')
        tf_y = tf.placeholder(tf.float32,
                    shape=(self.batch_size),
                    name='tf_y')
        tf_keepprob = tf.placeholder(tf.float32,
                    name='tf_keepprob')
        ## Create the embedding layer
        embedding = tf.Variable(
                    tf.random_uniform(
                        (self.n_words, self.embed_size),
                        minval=-1, maxval=1),
                    name='embedding')
        embed_x = tf.nn.embedding_lookup(
                    embedding, tf_x, 
                    name='embeded_x')

        ## Define LSTM cell and stack them together
        cells = tf.contrib.rnn.MultiRNNCell(
                [tf.contrib.rnn.DropoutWrapper(
                   tf.contrib.rnn.BasicLSTMCell(self.lstm_size),
                   output_keep_prob=tf_keepprob)
                 for i in range(self.num_layers)])

        ## Define the initial state:
        self.initial_state = cells.zero_state(
                 self.batch_size, tf.float32)
        print('  << initial state >> ', self.initial_state)

        lstm_outputs, self.final_state = tf.nn.dynamic_rnn(
                 cells, embed_x,
                 initial_state=self.initial_state)
        ## Note: lstm_outputs shape: 
        ##  [batch_size, max_time, cells.output_size]
        print('\n  << lstm_output   >> ', lstm_outputs)
        print('\n  << final state   >> ', self.final_state)

        ## Apply a FC layer after on top of RNN output:
        logits = tf.layers.dense(
                 inputs=lstm_outputs[:, -1],
                 units=1, activation=None,
                 name='logits')
        
        logits = tf.squeeze(logits, name='logits_squeezed')
        print ('\n  << logits        >> ', logits)
        
        y_proba = tf.nn.sigmoid(logits, name='probabilities')
        predictions = {
            'probabilities': y_proba,
            'labels' : tf.cast(tf.round(y_proba), tf.int32,
                 name='labels')
        }
        print('\n  << predictions   >> ', predictions)

        ## Define the cost function
        cost = tf.reduce_mean(
                 tf.nn.sigmoid_cross_entropy_with_logits(
                 labels=tf_y, logits=logits),
                 name='cost')
        
        ## Define the optimizer
        optimizer = tf.train.AdamOptimizer(self.learning_rate)
        train_op = optimizer.minimize(cost, name='train_op')

    def train(self, X_train, y_train, num_epochs):
        with tf.Session(graph=self.g) as sess:
            sess.run(self.init_op)
            iteration = 1
            for epoch in range(num_epochs):
                state = sess.run(self.initial_state)
                
                for batch_x, batch_y in create_batch_generator(
                            X_train, y_train, self.batch_size):
                    feed = {'tf_x:0': batch_x,
                            'tf_y:0': batch_y,
                            'tf_keepprob:0': 0.5,
                            self.initial_state : state}
                    loss, _, state = sess.run(
                            ['cost:0', 'train_op', 
                             self.final_state],
                            feed_dict=feed)

                    if iteration % 20 == 0:
                        print("Epoch: %d/%d Iteration: %d "
                              "| Train loss: %.5f" % (
                               epoch + 1, num_epochs,
                               iteration, loss))

                    iteration +=1
                if (epoch+1)%10 == 0:
                    self.saver.save(sess,
                        "model/sentiment-%d.ckpt" % epoch)

    def predict(self, X_data, return_proba=False):
        preds = []
        with tf.Session(graph = self.g) as sess:
            self.saver.restore(
                sess, tf.train.latest_checkpoint('model/'))
            test_state = sess.run(self.initial_state)
            for ii, batch_x in enumerate(
                create_batch_generator(
                    X_data, None, batch_size=self.batch_size), 1):
                feed = {'tf_x:0' : batch_x,
                        'tf_keepprob:0': 1.0,
                        self.initial_state : test_state}
                if return_proba:
                    pred, test_state = sess.run(
                        ['probabilities:0', self.final_state],
                        feed_dict=feed)
                else:
                    pred, test_state = sess.run(
                        ['labels:0', self.final_state],
                        feed_dict=feed)
                    
                preds.append(pred)
                
        return np.concatenate(preds)

Step 1: Defining multilayer RNN cells

Step 2: Defining the initial states for the RNN cells

Step 3: Creating the recurrent neural network using the RNN cells and their states

In [27]:
## Train:

n_words = max(list(word_to_int.values())) + 1

rnn = SentimentRNN(n_words=n_words, 
                   seq_len=sequence_length,
                   embed_size=256, 
                   lstm_size=128, 
                   num_layers=1, 
                   batch_size=15,# must divide evenly into the testing set number of samples
                   learning_rate=0.001)
  << initial state >>  (LSTMStateTuple(c=<tf.Tensor 'MultiRNNCellZeroState/DropoutWrapperZeroState/BasicLSTMCellZeroState/zeros:0' shape=(15, 128) dtype=float32>, h=<tf.Tensor 'MultiRNNCellZeroState/DropoutWrapperZeroState/BasicLSTMCellZeroState/zeros_1:0' shape=(15, 128) dtype=float32>),)

  << lstm_output   >>  Tensor("rnn/transpose_1:0", shape=(15, 200, 128), dtype=float32)

  << final state   >>  (LSTMStateTuple(c=<tf.Tensor 'rnn/while/Exit_3:0' shape=(15, 128) dtype=float32>, h=<tf.Tensor 'rnn/while/Exit_4:0' shape=(15, 128) dtype=float32>),)

  << logits        >>  Tensor("logits_squeezed:0", shape=(15,), dtype=float32)

  << predictions   >>  {'probabilities': <tf.Tensor 'probabilities:0' shape=(15,) dtype=float32>, 'labels': <tf.Tensor 'labels:0' shape=(15,) dtype=int32>}
In [28]:
rnn.train(X_train, y_train, num_epochs=40)
Epoch: 1/40 Iteration: 20 | Train loss: 0.09227
Epoch: 2/40 Iteration: 40 | Train loss: 0.02640
Epoch: 3/40 Iteration: 60 | Train loss: 0.07121
Epoch: 4/40 Iteration: 80 | Train loss: 0.19046
Epoch: 5/40 Iteration: 100 | Train loss: -0.77173
Epoch: 6/40 Iteration: 120 | Train loss: 0.00187
Epoch: 7/40 Iteration: 140 | Train loss: 0.00453
Epoch: 7/40 Iteration: 160 | Train loss: -1.20042
Epoch: 8/40 Iteration: 180 | Train loss: -2.62425
Epoch: 9/40 Iteration: 200 | Train loss: 0.00098
Epoch: 10/40 Iteration: 220 | Train loss: -1.81565
Epoch: 11/40 Iteration: 240 | Train loss: 0.00053
Epoch: 12/40 Iteration: 260 | Train loss: 0.00041
Epoch: 13/40 Iteration: 280 | Train loss: 0.00021
Epoch: 14/40 Iteration: 300 | Train loss: 0.00129
Epoch: 14/40 Iteration: 320 | Train loss: -3.88801
Epoch: 15/40 Iteration: 340 | Train loss: 0.00015
Epoch: 16/40 Iteration: 360 | Train loss: 0.00028
Epoch: 17/40 Iteration: 380 | Train loss: -2.39980
Epoch: 18/40 Iteration: 400 | Train loss: 0.00013
Epoch: 19/40 Iteration: 420 | Train loss: 0.00002
Epoch: 20/40 Iteration: 440 | Train loss: 0.00001
Epoch: 20/40 Iteration: 460 | Train loss: 0.00016
Epoch: 21/40 Iteration: 480 | Train loss: 0.00005
Epoch: 22/40 Iteration: 500 | Train loss: 0.00006
Epoch: 23/40 Iteration: 520 | Train loss: 0.00003
Epoch: 24/40 Iteration: 540 | Train loss: 0.00004
Epoch: 25/40 Iteration: 560 | Train loss: -2.39212
Epoch: 26/40 Iteration: 580 | Train loss: 0.00003
Epoch: 27/40 Iteration: 600 | Train loss: 0.00013
Epoch: 27/40 Iteration: 620 | Train loss: -3.00030
Epoch: 28/40 Iteration: 640 | Train loss: -5.58628
Epoch: 29/40 Iteration: 660 | Train loss: 0.00002
Epoch: 30/40 Iteration: 680 | Train loss: -3.29144
Epoch: 31/40 Iteration: 700 | Train loss: 0.00003
Epoch: 32/40 Iteration: 720 | Train loss: 0.00000
Epoch: 33/40 Iteration: 740 | Train loss: 0.00000
Epoch: 34/40 Iteration: 760 | Train loss: 0.00002
Epoch: 34/40 Iteration: 780 | Train loss: -6.43009
Epoch: 35/40 Iteration: 800 | Train loss: 0.00000
Epoch: 36/40 Iteration: 820 | Train loss: 0.00002
Epoch: 37/40 Iteration: 840 | Train loss: -3.07358
Epoch: 38/40 Iteration: 860 | Train loss: 0.00000
Epoch: 39/40 Iteration: 880 | Train loss: 0.00000
Epoch: 40/40 Iteration: 900 | Train loss: 0.00000
Epoch: 40/40 Iteration: 920 | Train loss: 0.00000
In [29]:
X_test.shape
Out[29]:
(150, 200)
In [30]:
y_test
Out[30]:
array([0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
       1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int64)
In [31]:
y_test.shape
Out[31]:
(150,)
In [32]:
X_test
Out[32]:
array([[ 0,  0,  0, ..., 13, 17, 22],
       [ 0,  0,  0, ..., 32,  4, 28],
       [ 0,  0,  0, ..., 26, 17, 22],
       ...,
       [ 0,  0,  0, ...,  4, 10,  1],
       [ 0,  0,  0, ...,  4, 10,  1],
       [ 0,  0,  0, ...,  4, 10,  1]])
In [33]:
X_test.shape[0]
Out[33]:
150
In [34]:
## Test: 
preds = rnn.predict(X_test)
y_true = y_test[:len(preds)]
print('Test Acc.: %.3f' % (
      np.sum(preds == y_true) / len(y_true)))
Test Acc.: 0.993
In [35]:
preds
Out[35]:
array([0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
       1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
In [36]:
preds.shape
Out[36]:
(150,)
In [37]:
prd = pd.DataFrame(preds)
prd.columns=['Predicted']
type(prd)
prd
Out[37]:
Predicted
0 0
1 0
2 1
3 0
4 1
... ...
145 0
146 0
147 0
148 0
149 0

150 rows × 1 columns

In [39]:
true = pd.DataFrame(y_test)
true.columns=['trueValue']
prd.index=true.index
pred=pd.concat([pd.DataFrame(prd),true],axis=1)
print(pred)
     Predicted  trueValue
0            0          0
1            0          0
2            1          1
3            0          0
4            1          1
..         ...        ...
145          0          0
146          0          0
147          0          0
148          0          0
149          0          0

[150 rows x 2 columns]
In [40]:
## Get probabilities:
proba = rnn.predict(X_test, return_proba=True)
In [41]:
proba
Out[41]:
array([6.64591789e-06, 2.44379044e-06, 1.00000000e+00, 1.19209290e-06,
       1.00000000e+00, 3.41236591e-05, 9.03010368e-06, 1.78813934e-07,
       1.07288361e-06, 1.07288361e-06, 1.00000000e+00, 1.07288361e-06,
       1.01133787e-06, 1.01133787e-06, 1.01133685e-06, 1.01327896e-06,
       9.53674316e-07, 1.00000000e+00, 1.00000000e+00, 1.01327896e-06,
       1.01327896e-06, 1.01327896e-06, 1.00000000e+00, 1.07288361e-06,
       1.07288361e-06, 9.53674316e-07, 1.07288361e-06, 1.00000000e+00,
       1.01133685e-06, 1.01133594e-06, 1.07288361e-06, 1.07288361e-06,
       1.00000000e+00, 1.01327896e-06, 1.07288361e-06, 1.07288361e-06,
       1.07288361e-06, 1.01327896e-06, 1.07288361e-06, 1.00000000e+00,
       1.07288361e-06, 1.07288361e-06, 1.01177193e-06, 2.85027824e-07,
       1.01133594e-06, 1.07288361e-06, 1.07288361e-06, 9.53674316e-07,
       1.07288361e-06, 1.07288361e-06, 1.90734863e-06, 7.15255737e-07,
       7.15255737e-07, 7.15255737e-07, 7.74860382e-07, 7.15255737e-07,
       7.15255737e-07, 7.63040703e-07, 7.63041385e-07, 3.91007507e-06,
       1.07288361e-06, 1.00000000e+00, 1.00000000e+00, 1.78813934e-07,
       1.49011612e-07, 3.27825546e-06, 5.28991222e-05, 5.28991222e-05,
       5.26189804e-04, 2.47359276e-06, 1.56083405e-02, 1.05798244e-05,
       6.88267937e-06, 1.44621263e-05, 2.43651562e-06, 1.72257423e-04,
       1.49011612e-07, 4.70995903e-04, 1.72257423e-04, 1.00000000e+00,
       1.00000000e+00, 3.27825546e-06, 3.27825546e-06, 1.72257423e-04,
       6.55651093e-07, 2.68220901e-07, 3.57627869e-07, 3.67640041e-07,
       7.84295580e-08, 7.84295580e-08, 8.94069672e-08, 8.94069672e-08,
       1.00000000e+00, 8.94069672e-08, 8.94069672e-08, 1.49011612e-07,
       1.49011612e-07, 1.49011612e-07, 8.94069672e-08, 4.26173210e-06,
       8.94069672e-08, 8.94069672e-08, 7.84295580e-08, 7.84297072e-08,
       1.00000000e+00, 2.23517418e-06, 5.96046448e-08, 5.96046448e-08,
       8.94069672e-08, 8.94069672e-08, 8.94069672e-08, 8.94069672e-08,
       8.94069672e-08, 8.94069672e-08, 1.49011612e-07, 5.96046448e-08,
       5.96046448e-08, 7.84298564e-08, 7.84297072e-08, 7.84584273e-08,
       5.96046448e-08, 8.94069672e-08, 8.94069672e-08, 8.94069672e-08,
       5.96046448e-08, 8.94069672e-08, 1.00000000e+00, 1.00000000e+00,
       8.94069672e-08, 8.94069672e-08, 8.94069672e-08, 8.94069672e-08,
       7.84298564e-08, 7.84297072e-08, 7.84298564e-08, 8.94069672e-08,
       8.94069672e-08, 8.94069672e-08, 8.94069672e-08, 8.94069672e-08,
       8.94069672e-08, 8.94069672e-08, 1.49011612e-07, 8.94069672e-08,
       8.94069672e-08, 5.96046448e-08, 5.96046448e-08, 7.84298564e-08,
       7.84297072e-08, 7.84298564e-08], dtype=float32)
In [42]:
proba.shape
Out[42]:
(150,)

Example application: character-level language modeling

In [13]:
Image(filename='images/16_11.png', width=600) 
Out[13]:

Preparing the data

In [14]:
Image(filename='images/16_12.png', width=600) 
Out[14]:
In [15]:
Image(filename='images/16_13.png', width=600) 
Out[15]:
In [16]:
Image(filename='images/16_14.png', width=600) 
Out[16]:
In [43]:
import numpy as np


## Reading and processing text
with open('2columnsNunez.txt', 'r', encoding='utf-8') as f: 
    text=f.read()
In [44]:
text
Out[44]:
"missed R mt kick to low L leg\nmissed L jab\nmissed R mt kick to low L leg\nlands L mt kick to L leg\nmissed L jab, missed R cross\nlands L mt kick to L leg, missed R cross to face\nholding R leg hold starts\nholding R leg hold continues\nholding R leg hold continues\nholding R leg hold continues misses R cross to face\nloses R leg hold\nlands R mt kick to L low leg\nmisses L jab to face, misses R cross to face\nmisses L jab to face, misses R cross to face\nmisses L jab to face \nattempts body waste takedown\nlands takedown\ncaught in full guard hold\ncaught in full guard hold\ncaught in full guard hold\ncaught in full guard hold\ncaught in full guard hold\ncaught in full guard hold and puts weight on L shoulder collarbone acromian process while lifting off L leg to break full guard\ncaught in full guard hold\ncaught in full guard hold and twisting with R armpit and R leg anchoring into L locked thigh at the inside knee\ncaught in full guard hold and grabbing at L ear with R hand while twisting and lifting out of full guard lock, gains L arm back, breaks full guard lock at ankles\nbreaks full guard hold and standing up while keeping L shin of opponent at chest and bent\nholding L knee hold starts of opponent while standing up and maneuvering around L side of opponent pushing forward to knock off balance a possible up kick of opponent\ncaught in open guard hold continues and misses downward R hook to head while pushing and twisting L and holding L knee hold continues\ncaught in open guard hold continues and lands downward R cross to head while pushing opponent's leg toward her and turning L and loses L knee hold\ncaught in open guard hold continues and lands L downward cross to face while getting out of open guard completely\ncaught in open guard hold continues  \ncaught in open guard hold continues and holding R foot while opponent in open guard\ncaught in open guard hold contnues and moves open guard feet to her R while landing R downward cross to face\nbreaks open guard hold and falls on ground to the side of opponent\ngetting up while attempting a head R arm guillotine of opponent as holding guillotine hold starts\nlocks guillotine around R armpit and neck of opponent while standing full guard lock and holding guillotine hold continues\nholding guillotine hold continues\nholding guillotine hold continues\ntakedown of opponent in full guard while holding of guillotine hold continues\nholding guillotine hold continues\nholding full guard hold starts and holding guillotine hold continues\nholding guillotine hold continues\nholding guillotine hold continues\nholding guillotine hold continues\nloses guillotine hold and loses full guard hold\npush kicks opponent off \ngetting up while L foot being held by opponent\ngetting up from ground\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\ncaught in R arm hold and caught in upper body hold continues and moves opponent away from side standing control against cage\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nholding R arm hold starts and caught in upper body hold\nloses R arm hold and caught in upper body hold\nholding neck hold starts and caught in upper body hold\nbreaks upper body hold to switch sides\ncaught in upper body hold and holding R arm hold starts\nbreaks upper body hold and loses R arm hold and lands a R leg to L leg of opponent sweeping takedown\nlands slight double leg takedown\nnot on the ground but leaning over opponent\ncaught in open guard hold  and lands downward R cross to face, lands downward L cross to face, lands downward R cross to face\ncaught in open guard hold  and lands downward R cross to face\ncaught in open guard hold  and lands downward R cross to face\ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold  and breaks into opponents open guard hold  against cage and lands R downward cross to head\ncaught in open guard hold  and lands downward R cross to head, lands downward R cross to head\ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold  and lands R downward cross to head\ncaught in open guard hold  and lands R downward cross to head\ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold  and twists L forearm of opponent at the elbow pushing the hand  in open guard hold  from standing leaning over position\ncaught in open guard hold \ncaught in open guard hold \nbreaks open guard hold  hold\ncaught in full guard hold \ncaught in full guard hold \ncaught in full guard hold \ncaught in full guard hold \ncaught in full guard hold \ncaught in full guard hold \ncaught in full guard hold \nlands R elbow to face and caught in full guard hold \ncaught in full guard hold \ncaught in full guard hold \ncaught in full guard hold \ncaught in full guard hold \nbreaks full guard hold  and caught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\nlands R downward cross to face and caught in open guard hold\ncaught in open guard hold\nlands R downward cross to face and caught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\ncaught in open guard hold and misses L downward cross to head, lands R downward cross to head\ncaught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\nbreaks through leg kicks of open guard and caught in open guard hold\nlands downward R cross to head and caught in open guard hold\nlands downward R cross to head and caught in open guard hold\nlands downward R cross to head and caught in open guard hold, lands downward R cross to head\nlands downward R cross to head and caught in open guard hold, lands downward R cross to head\nmisses downward R cross to head and caught in open guard hold, misses downward R cross to head\nlands downward L cross to head and caught in open guard hold, lands downward L cross to head\nlands downward R cross to head and caught in open guard hold\nmisses downward R cross to head and caught in open guard hold\ncaught in open guard hold\ncaught in open guard hold\nlands downward R cross to head and caught in open guard hold, lands downward L cross to head\nmisses downward R cross to head and caught in open guard hold, misses downward L cross to head, lands L downward cross to head\nlands downward R cross to head and caught in open guard hold\nlands downward R cross to head and caught in open guard hold, lands downward L cross to head\nlands downward R cross to head and caught in open guard hold\nlands downward L cross to head and caught in open guard hold\nmisses downward R cross to head and caught in open guard hold, lands L downward cross to head\nlands downward R cross to head and caught in open guard hold, lands L downward cross to head\nmisses downward R cross to head and caught in open guard hold, misses downward L cross to head\nmisses downward R cross to head and caught in open guard hold, misses downward L cross to head, misses downward R cross to head\npressing opponent into ground from the top at opponents shoulders and neck having open guard and legs up off balance removing leverage to kick up and caught in open guard hold\ncaught in open guard hold\nknee pinned to abdomen  and holding L side control hold starts and breaks open guard hold\nholding side control hold continues\nlands L downward cross to head and holding side control hold continues\nlands L downward hook to face and holding side control hold continues, lands L downward elbow to face\nholding side control hold continues\nholding side control hold continues\nholding side control hold continues\nholding side control hold continues\nholding side control hold continues\nL back side control hold with L knee pinned to opponent's L hip and holding side control hold continues\nholding side control hold continues\nholding side control hold continues\nholding side control hold continues\nholding side control hold continues\nholding side control hold continues\nholding side control hold continues\nholding side control hold continues\nholding side control hold continues\nholding a L armpit and neck choke hold starts and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nholding choke hold continues and holding side control hold continues\nloses choke hold and holding side control hold continues and pins L knee on opponent's abdomen\nmisses R hammer to face  holding side control hold continues with knee pinning opponent's abdomen\n holding side control hold continues\nlands L downward elbow to the face from pinned side control mount\nmisses L downward elbow to face\nloses side control hold\nmisses downward R cross to face\ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \nmisses downward flying R cross to face and caught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \ncaught in open guard hold \nround ends caught in open guard hold\nmisses L jab\nmisses R mt kick to low L lead leg\nmisses R mt kick to low L lead leg\nmisses L jab\nmisses R mt kick to low L lead leg\ngets up out of takedown\nmisses R cross to face\nmisses R mt kick to low L lead leg\nmisses R cross to face\nmisses R cross to face\nlands body throw takedown gets up\nmisses illegal right knee to the face\nmisses L jab\nmisses L jab, misses R cross to face\nmisses R cross to face\nmisses L jab\nlands R cross\nmisses R cross to face\nmisses R cross to face\nmisses R cross to face\nmisses L jab to face, ducks to avoid body takedown from missed L jab\nmisses R cross to face\nmisses L jab\nlands R cross to face\nmisses R cross to face\nlands L hook to face\nmisses R cross to face\nlands R cross\nlands L upper to face\nmisses R upper to face, misses R upper to face\nlands L knee to face\nmisses L jab, misses R cross to face\nlands L jab to face\nmisses R cross to face\nmisses R mt kick to inside R leg\nlands L jab to face, lands R cross to face\nmisses R cross to face\nlands 2 hammer hits R face on top back\nlands 2 hammer hits L face on top back\nlands 2 hammer hits L face on top back\nlands RNC\nflips on back w RNC\nmisses R mt kick to low L lead leg\nmisses R cross\nmisses R mt kick to low L lead leg\nmisses R mt kick to low L lead leg\nmisses R mt kick to low L lead leg\nmisses L jab to face\nmisses R mt kick to low L lead leg\nfeigns L mt kick to inside L lead leg\nmissses R mt kick to low L lead leg\nmisses R mt kick to low L lead leg\nlands L jab to body, misses R cross to face\nmisses L jab to body\nmisses R mt kick to low L lead leg\nmisses R mt kick to low L lead leg\nmisses R mt kick to low L lead leg\nmisses L jab to body\nmisses R mt kick to low L lead leg\nmisses L jab to face, misses R cross to face\npushes off with R forearm\nmisses L jab to body\nlands R mt kick to low L back of knee, misses L jab, misses R cross\nblocks L jab by raising L Shoulder, misses R cross to face\npushes Face with R hand and away\nmisses L jab to body\nmisses L jab to body, leans back from R cross to face\nmisses R cross\nmisses L jab to face\nmisses L jab to body\nmisses R cross to face\nmisses L jab to face\nducks \nmisses R mt kick to low L lead leg\nducks and clinches waist pushes to cage\nlands R upper to face \nmisses R cross to face\nlands R mt kick to low L back of knee\nblocks L hook with R overhead arm\nmisses R mt kick to low L lead leg\nlands R push kick to body\nmisses L jab, misses R cross to face\nmisses R push kick to body\nmisses L jab to face\nmisses R cross to face, misses R cross to face\nmisses R mt kick to low L lead leg\nmisses R cross to face\nmisses R mt kick to body\nducks\nmisses R upper to face\nmisses L jab, misses R cross to face\nmisses L jab to body\nfeigns L mt kick to head\nmisses L jab, misses R cross to face\nmisses R mt kick to low L lead leg\nmisses R cross to face\nmisses L jab, misses R cross to face\nmisses L mt kick to inside L lead leg\nlands L knee to body with neck clinch\nmisses L jab to face\nmisses R mt kick to low L lead leg\nmisses R cross to face\nmisses R cross\nmisses L jab\nmisses R push kick to body\nmissed a L jab to face\nmissed a R cross to face\nmissed a R cross to face\nmissed a R push kick to body\nmissed a R cross to face, lands a L jab to face\nmissed a L jab to face, lands a R cross to face \nmissed a R cross, missed a L jab to face, missed a R cross to face\nmissed a R push kick to body\nlands a R cross to face\nturns body to side gets out of clinch\nlands L jab to face, misses R cross to face\nmisses L jab to face, misses R cross to face, \nlands L jab to face face\nmisses R cross to face, misses L upper\nmisses R cross to body\nlands L jab to face\nlands R cross to face, ducks R cross left\nmisses L jab to face, misses R cross to face, \nmisses R upper to face\nlands L jab to face, misses R cross to face\nmisses L jab to face, lands R cross to face\nruns to and feigns a R cross to face\nlands a R cross to face\nmisses R cross to face \nlands a R cross to face\nlands a L jab to face, lands R cross to face\nlands downward L jab to face\nreferee gets between and stops fight TKO\n"
In [45]:
chars = set(text)
char2int = {ch:i for i,ch in enumerate(chars)}
int2char = dict(enumerate(chars))
text_ints = np.array([char2int[ch] for ch in text], 
                     dtype=np.int32)
In [ ]:
## @Readers: PLEASE IGNORE THIS CELL
##
## This cell is meant to shrink the
## dataset when this notebook is run 
## on the Travis Continuous Integration
## platform to test the code as well as
## speeding up the run using a smaller
## dataset for debugging

if 'TRAVIS' in os.environ:
    text = text[:1000]
    chars = set(text)
    char2int = {ch:i for i,ch in enumerate(chars)}
    int2char = dict(enumerate(chars))
    text_ints = np.array([char2int[ch] for ch in text], 
                         dtype=np.int32)
In [46]:
def reshape_data(sequence, batch_size, num_steps):
    tot_batch_length = batch_size * num_steps
    num_batches = int(len(sequence) / tot_batch_length)
    if num_batches*tot_batch_length + 1 > len(sequence):
        num_batches = num_batches - 1
    ## Truncate the sequence at the end to get rid of 
    ## remaining charcaters that do not make a full batch
    x = sequence[0 : num_batches*tot_batch_length]
    y = sequence[1 : num_batches*tot_batch_length + 1]
    ## Split x & y into a list batches of sequences: 
    x_batch_splits = np.split(x, batch_size)
    y_batch_splits = np.split(y, batch_size)
    ## Stack the batches together
    ## batch_size x tot_batch_length
    x = np.stack(x_batch_splits)
    y = np.stack(y_batch_splits)
    
    return x, y

## Testing:
train_x, train_y = reshape_data(text_ints, 64, 10)
print(train_x.shape)
print(train_x[0, :10])
print(train_y[0, :10])
print(''.join(int2char[i] for i in train_x[0, :50]))
(64, 240)
[11 36 31 31  1 21 23  6 23 11]
[36 31 31  1 21 23  6 23 11 10]
missed R mt kick to low L leg
missed L jab
missed 
In [47]:
np.random.seed(123)

def create_batch_generator(data_x, data_y, num_steps):
    batch_size, tot_batch_length = data_x.shape    
    num_batches = int(tot_batch_length/num_steps)
    for b in range(num_batches):
        yield (data_x[:, b*num_steps: (b+1)*num_steps], 
               data_y[:, b*num_steps: (b+1)*num_steps])
        
bgen = create_batch_generator(train_x[:,:100], train_y[:,:100], 15)
for b in bgen:
    print(b[0].shape, b[1].shape, end='  ')
    print(''.join(int2char[i] for i in b[0][0,:]).replace('\n', '*'), '    ',
          ''.join(int2char[i] for i in b[1][0,:]).replace('\n', '*'))
(64, 15) (64, 15)  missed R mt kic      issed R mt kick
(64, 15) (64, 15)  k to low L leg*       to low L leg*m
(64, 15) (64, 15)  missed L jab*mi      issed L jab*mis
(64, 15) (64, 15)  ssed R mt kick       sed R mt kick t
(64, 15) (64, 15)  to low L leg*la      o low L leg*lan
(64, 15) (64, 15)  nds L mt kick t      ds L mt kick to

Building the character-level RNN model

In [48]:
import tensorflow as tf
import os

class CharRNN(object):
    def __init__(self, num_classes, batch_size=64, 
                 num_steps=100, lstm_size=128, 
                 num_layers=1, learning_rate=0.001, 
                 keep_prob=0.5, grad_clip=5, 
                 sampling=False):
        self.num_classes = num_classes
        self.batch_size = batch_size
        self.num_steps = num_steps
        self.lstm_size = lstm_size
        self.num_layers = num_layers
        self.learning_rate = learning_rate
        self.keep_prob = keep_prob
        self.grad_clip = grad_clip
        
        self.g = tf.Graph()
        with self.g.as_default():
            tf.set_random_seed(123)

            self.build(sampling=sampling)
            self.saver = tf.train.Saver()
            self.init_op = tf.global_variables_initializer()
            
    def build(self, sampling):
        if sampling == True:
            batch_size, num_steps = 1, 1
        else:
            batch_size = self.batch_size
            num_steps = self.num_steps

        tf_x = tf.placeholder(tf.int32, 
                              shape=[batch_size, num_steps], 
                              name='tf_x')
        tf_y = tf.placeholder(tf.int32, 
                              shape=[batch_size, num_steps], 
                              name='tf_y')
        tf_keepprob = tf.placeholder(tf.float32, 
                              name='tf_keepprob')

        # One-hot encoding:
        x_onehot = tf.one_hot(tf_x, depth=self.num_classes)
        y_onehot = tf.one_hot(tf_y, depth=self.num_classes)

        ### Build the multi-layer RNN cells
        cells = tf.contrib.rnn.MultiRNNCell(
            [tf.contrib.rnn.DropoutWrapper(
                tf.contrib.rnn.BasicLSTMCell(self.lstm_size), 
                output_keep_prob=tf_keepprob) 
            for _ in range(self.num_layers)])
        
        ## Define the initial state
        self.initial_state = cells.zero_state(
                    batch_size, tf.float32)

        ## Run each sequence step through the RNN 
        lstm_outputs, self.final_state = tf.nn.dynamic_rnn(
                    cells, x_onehot, 
                    initial_state=self.initial_state)
        
        print('  << lstm_outputs  >>', lstm_outputs)

        seq_output_reshaped = tf.reshape(
                    lstm_outputs, 
                    shape=[-1, self.lstm_size],
                    name='seq_output_reshaped')

        logits = tf.layers.dense(
                    inputs=seq_output_reshaped, 
                    units=self.num_classes,
                    activation=None,
                    name='logits')

        proba = tf.nn.softmax(
                    logits, 
                    name='probabilities')
        print(proba)

        y_reshaped = tf.reshape(
                    y_onehot, 
                    shape=[-1, self.num_classes],
                    name='y_reshaped')
        cost = tf.reduce_mean(
                    tf.nn.softmax_cross_entropy_with_logits(
                        logits=logits, 
                        labels=y_reshaped),
                    name='cost')

        # Gradient clipping to avoid "exploding gradients"
        tvars = tf.trainable_variables()
        grads, _ = tf.clip_by_global_norm(
                    tf.gradients(cost, tvars), 
                    self.grad_clip)
        optimizer = tf.train.AdamOptimizer(self.learning_rate)
        train_op = optimizer.apply_gradients(
                    zip(grads, tvars),
                    name='train_op')
        
    def train(self, train_x, train_y, 
              num_epochs, ckpt_dir='./model/'):
        ## Create the checkpoint directory
        ## if does not exists
        if not os.path.exists(ckpt_dir):
            os.mkdir(ckpt_dir)
            
        with tf.Session(graph=self.g) as sess:
            sess.run(self.init_op)

            n_batches = int(train_x.shape[1]/self.num_steps)
            iterations = n_batches * num_epochs
            for epoch in range(num_epochs):

                # Train network
                new_state = sess.run(self.initial_state)
                loss = 0
                ## Minibatch generator:
                bgen = create_batch_generator(
                        train_x, train_y, self.num_steps)
                for b, (batch_x, batch_y) in enumerate(bgen, 1):
                    iteration = epoch*n_batches + b
                    
                    feed = {'tf_x:0': batch_x,
                            'tf_y:0': batch_y,
                            'tf_keepprob:0': self.keep_prob,
                            self.initial_state : new_state}
                    batch_cost, _, new_state = sess.run(
                            ['cost:0', 'train_op', 
                                self.final_state],
                            feed_dict=feed)
                    if iteration % 10 == 0:
                        print('Epoch %d/%d Iteration %d'
                              '| Training loss: %.4f' % (
                              epoch + 1, num_epochs, 
                              iteration, batch_cost))

                ## Save the trained model    
                self.saver.save(
                        sess, os.path.join(
                            ckpt_dir, 'language_modeling.ckpt'))
                              
                              
                
    def sample(self, output_length, 
               ckpt_dir, starter_seq="The "):
        observed_seq = [ch for ch in starter_seq]        
        with tf.Session(graph=self.g) as sess:
            self.saver.restore(
                sess, 
                tf.train.latest_checkpoint(ckpt_dir))
            ## 1: run the model using the starter sequence
            new_state = sess.run(self.initial_state)
            for ch in starter_seq:
                x = np.zeros((1, 1))
                x[0,0] = char2int[ch]
                feed = {'tf_x:0': x,
                        'tf_keepprob:0': 1.0,
                        self.initial_state: new_state}
                proba, new_state = sess.run(
                        ['probabilities:0', self.final_state], 
                        feed_dict=feed)

            ch_id = get_top_char(proba, len(chars))
            observed_seq.append(int2char[ch_id])
            
            ## 2: run the model using the updated observed_seq
            for i in range(output_length):
                x[0,0] = ch_id
                feed = {'tf_x:0': x,
                        'tf_keepprob:0': 1.0,
                        self.initial_state: new_state}
                proba, new_state = sess.run(
                        ['probabilities:0', self.final_state], 
                        feed_dict=feed)

                ch_id = get_top_char(proba, len(chars))
                observed_seq.append(int2char[ch_id])

        return ''.join(observed_seq)
In [49]:
def get_top_char(probas, char_size, top_n=5):
    p = np.squeeze(probas)
    p[np.argsort(p)[:-top_n]] = 0.0
    p = p / np.sum(p)
    ch_id = np.random.choice(char_size, 1, p=p)[0]
    return ch_id
In [50]:
batch_size = 64
num_steps = 100 
train_x, train_y = reshape_data(text_ints, 
                                batch_size, 
                                num_steps)

rnn = CharRNN(num_classes=len(chars), batch_size=batch_size)
rnn.train(train_x, train_y, 
          num_epochs=100,
          ckpt_dir='./model-100/')
W0528 09:40:27.499595  6288 deprecation.py:323] From <ipython-input-48-ec75d21cdc9c>:88: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See `tf.nn.softmax_cross_entropy_with_logits_v2`.

  << lstm_outputs  >> Tensor("rnn/transpose_1:0", shape=(64, 100, 128), dtype=float32)
Tensor("probabilities:0", shape=(6400, 37), dtype=float32)
Epoch 5/100 Iteration 10| Training loss: 3.2700
Epoch 10/100 Iteration 20| Training loss: 3.0537
Epoch 15/100 Iteration 30| Training loss: 2.9880
Epoch 20/100 Iteration 40| Training loss: 2.9461
Epoch 25/100 Iteration 50| Training loss: 2.9077
Epoch 30/100 Iteration 60| Training loss: 2.8788
Epoch 35/100 Iteration 70| Training loss: 2.8224
Epoch 40/100 Iteration 80| Training loss: 2.7633
Epoch 45/100 Iteration 90| Training loss: 2.6782
Epoch 50/100 Iteration 100| Training loss: 2.5800
Epoch 55/100 Iteration 110| Training loss: 2.4614
Epoch 60/100 Iteration 120| Training loss: 2.3618
Epoch 65/100 Iteration 130| Training loss: 2.2528
Epoch 70/100 Iteration 140| Training loss: 2.1530
Epoch 75/100 Iteration 150| Training loss: 2.0454
Epoch 80/100 Iteration 160| Training loss: 1.9497
Epoch 85/100 Iteration 170| Training loss: 1.8562
Epoch 90/100 Iteration 180| Training loss: 1.7785
Epoch 95/100 Iteration 190| Training loss: 1.6990
Epoch 100/100 Iteration 200| Training loss: 1.6294
In [52]:
np.random.seed(123)
rnn = CharRNN(len(chars), sampling=True)

print(rnn.sample(ckpt_dir='./model-100/', 
                 output_length=500))
  << lstm_outputs  >> Tensor("rnn/transpose_1:0", shape=(1, 1, 128), dtype=float32)
Tensor("probabilities:0", shape=(1, 37), dtype=float32)
The holdnnis  andd aad ho sn  aad caurd h hld  ugadi non  auuhhd nnngauad ahd cant in uenn old iuad  oold
caughti  oopenngaghd hlnd nard hnld nnt inuuhodhond nonisn ndanndin sos hoed caadh hodin guue ahld onng audtin ne aand hon  nuaed cant inue inodinghol  opdne aond ooldgnued ollndiuest in  olding holeno ugaud hold gugurt iool ncguhuhld naan  aad  hudin uuad  aodd aad h ad ing gaur hold aoldinn  aard hodl
nntuas  aoldnnughari ntoed cagtht nnlenguuaa  h lldnguauh dholn guard oald ant s ne ngaah in o
In [53]:
## run for 200 epochs
batch_size = 64
num_steps = 100 

rnn = CharRNN(num_classes=len(chars), batch_size=batch_size)
rnn.train(train_x, train_y, 
          num_epochs=200,
          ckpt_dir='./model-200/')
  << lstm_outputs  >> Tensor("rnn/transpose_1:0", shape=(64, 100, 128), dtype=float32)
Tensor("probabilities:0", shape=(6400, 37), dtype=float32)
Epoch 5/200 Iteration 10| Training loss: 3.2700
Epoch 10/200 Iteration 20| Training loss: 3.0537
Epoch 15/200 Iteration 30| Training loss: 2.9880
Epoch 20/200 Iteration 40| Training loss: 2.9461
Epoch 25/200 Iteration 50| Training loss: 2.9077
Epoch 30/200 Iteration 60| Training loss: 2.8788
Epoch 35/200 Iteration 70| Training loss: 2.8224
Epoch 40/200 Iteration 80| Training loss: 2.7633
Epoch 45/200 Iteration 90| Training loss: 2.6782
Epoch 50/200 Iteration 100| Training loss: 2.5800
Epoch 55/200 Iteration 110| Training loss: 2.4614
Epoch 60/200 Iteration 120| Training loss: 2.3618
Epoch 65/200 Iteration 130| Training loss: 2.2528
Epoch 70/200 Iteration 140| Training loss: 2.1530
Epoch 75/200 Iteration 150| Training loss: 2.0454
Epoch 80/200 Iteration 160| Training loss: 1.9497
Epoch 85/200 Iteration 170| Training loss: 1.8562
Epoch 90/200 Iteration 180| Training loss: 1.7785
Epoch 95/200 Iteration 190| Training loss: 1.6990
Epoch 100/200 Iteration 200| Training loss: 1.6294
Epoch 105/200 Iteration 210| Training loss: 1.5427
Epoch 110/200 Iteration 220| Training loss: 1.4891
Epoch 115/200 Iteration 230| Training loss: 1.4572
Epoch 120/200 Iteration 240| Training loss: 1.3827
Epoch 125/200 Iteration 250| Training loss: 1.3378
Epoch 130/200 Iteration 260| Training loss: 1.2864
Epoch 135/200 Iteration 270| Training loss: 1.2351
Epoch 140/200 Iteration 280| Training loss: 1.2003
Epoch 145/200 Iteration 290| Training loss: 1.1607
Epoch 150/200 Iteration 300| Training loss: 1.1145
Epoch 155/200 Iteration 310| Training loss: 1.0948
Epoch 160/200 Iteration 320| Training loss: 1.0551
Epoch 165/200 Iteration 330| Training loss: 1.0157
Epoch 170/200 Iteration 340| Training loss: 0.9881
Epoch 175/200 Iteration 350| Training loss: 0.9627
Epoch 180/200 Iteration 360| Training loss: 0.9402
Epoch 185/200 Iteration 370| Training loss: 0.9088
Epoch 190/200 Iteration 380| Training loss: 0.8854
Epoch 195/200 Iteration 390| Training loss: 0.8715
Epoch 200/200 Iteration 400| Training loss: 0.8345
In [54]:
del rnn

np.random.seed(123)
rnn = CharRNN(len(chars), sampling=True)
print(rnn.sample(ckpt_dir='./model-200/', 
                 output_length=500))
  << lstm_outputs  >> Tensor("rnn/transpose_1:0", shape=(1, 1, 128), dtype=float32)
Tensor("probabilities:0", shape=(1, 37), dtype=float32)
The holding hold cant hold nnts ane holding sadd contros hold and cantin  openeguard hold
mosssid R carrss old arad lnd gulr hol hold toniiues holding Ris tol on lland ldonngnding aued contiugs and ones and inden cruhh hold lndine sadd canttin ouguar  hold
canghe inn ole guarl hold
caught in  open guard hold
caught in opendguard hold
caught in open guard hold and ardss and L ard sown ard micess tol cnuts id coatsin  oeen anrdhhold contintss and hold nghindownin  hol  oll conestine 
cought in oen guar

Summary

...


Readers may ignore the next cell.

In [1]:
! python ../.convert_notebook_to_script.py --input ch16.ipynb --output ch16.py
[NbConvertApp] Converting notebook ch16.ipynb to script
[NbConvertApp] Writing 24796 bytes to ch16.py